Appl. No. 09/927,558 

Reply to the Office Action of November 14, 2007 



Amendments to the Claims: 

This listing of claims will replace all prior versions, and listings, of claims in the application: 

Listing of Claims: 

Please amend claims 1, 12, 18, and 29, as follows: 

1 . (Currently Amended) A method of recovering depth information for pixels of a 
base image representing a view of a scene, the method comprising the steps of: 

detecting a plurality of pixels in a base image that represents a first view of a scene, 
wherein the first view is one of at least three relative views of a scene; 

determining 3-D depth of the plurality of pixels in the base image by contemporaneously 
matching image features between three or more relative images of a scene by matching 
correspondence to a plurality of pixels in three or more relative images representing the at least 
three relative views of the scene, wherein the three or more images are captured relative to each 
other and are of the same scene, and wherein each of the three or more images includes at least 
an overlapping portion of the scene relative to each other, wherein the at least three relative 
views of the scene are situated in a non-linear arrangement relative to each other and are further 
oriented in non-parallel planes relative to each other; and 

tracing pixels in a virtual piecewise continuous depth surface by spatial propagation 
starting from the detected pixels in the base image by using the matching and corresponding 
plurality of pixels in the three or more relative images to create the virtual piecewise continuous 
depth surface viewed from the base image, each successfully traced pixel being associated with a 
depth in the scene viewed from the base image. 
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2. (Previously Presented) The method of Claim 1, wherein the step of determining the 3-D depth 
comprises the steps of: 

detecting a plurality of image pixels in a first image corresponding to a first view of a 

scene; 

detecting a plurality of image pixels in at least a second image corresponding to a 
respective at least a second view of the scene, wherein the at least a second image deviates from 
the first image as a result of camera relative motion; 

determining a first two-view correspondence between the plurality of detected image 
pixels in the first image and a plurality of detected image pixels in one of the at least a second 
image resulting in a potential two-view match set of candidate image pixels between the first 
image and the one of the at least a second image; 

concatenating a plurality of two-view match sets to form a first multiple-view potential 
match set; and 

determining a second multiple-view potential match set that is a refinement of the first 
multiple-view potential match set. 

3. (Previously Presented) The method of claim 2, wherein the second multiple- view potential 
match set is based at least in part on a least median of squares computation of the reprojection 
errors related to matched pixels in the first multiple- view potential match set. 

4. (Original) The method of Claim 1, wherein the tracing step comprises the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel starting from the detected pixels in the base image. 

5. (Original) The method of Claim 4, wherein the tracing step comprises the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 
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6. (Original) The method of Claim 4, wherein the tracing step comprises the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

7. (Original) The method of Claim 1, wherein the tracing step comprises the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel surrounded by a predefined size window in the continuous depth surface. 

8. (Original) The method of Claim 7, wherein the tracing step comprises the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 

9. (Original) The method of Claim 7, wherein the tracing step comprises the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

10. (Previously Presented) The method of Claim 9, wherein the matching cost is determined by 
computing the summation of all the normalized cross-correlations between a first image window 
of a pre-determined size in the base view, and a second image window of the same pre- 
determined size in one of at least one reference view. 

11. (Previously Presented) The method of Claim 10, further comprising a step of: 

rectification of at least one pair of images corresponding to the base view of the scene 
and one of the at least one reference view. 
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12. (Currently Amended) An image processing system comprising: 
a memory; 

a controller/processor electrically coupled to the memory; 

an image matching module, electrically coupled to the controller/processor and to the 
memory, for providing a plurality of seed pixels that represent 3-D depth of the plurality of 
pixels in the base image view of a scene by contemporaneously matching image features 
between three or more relative images of a scene by matching correspondence to a plurality of 
pixels in three or more relative images representing a plurality of relative views of the scene, 
wherein the three or more relative images are captured relative to each other and are of the same 
scene, and wherein each of the three or more images includes at least an overlapping portion of 
the scene relative to each other, wherein the base image view is one of at least three relative 
views of the scene, the at least three relative views of the scene being situated in a non-linear 
arrangement and are further oriented in non-parallel planes relative to each other; and 

a propagation module, electrically coupled to the controller/processor and to the memory, 
for tracing pixels in a virtual piecewise continuous depth surface by spatial propagation starting 
from the provided plurality of seed pixels in the base image by using the matching and 
corresponding plurality of pixels in the three or more relative images to create the virtual 
piecewise continuous depth surface viewed from the base image, each successfully traced pixel 
being associated with a depth in the scene viewed from the base image. 

13. (Original) The image processing system of claim 12, further comprising at least one camera 
interface, electrically coupled to the controller/processor, for sending image information from at 
least one camera to the controller/processor. 

14. (Original) The image processing system of claim 12, wherein the controller/processor, the 
memory, the image matching module, and the propagation module, are implemented in at least 
one of an integrated circuit, a circuit supporting substrate, and a scanner. 
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15. (Original) The image processing system of Claim 12, wherein the propagation module is 
further for determining when a boundary is reached between two propagating fronts of virtual 
pieces of a continuous depth surface. 

16. (Original) The image processing system of Claim 15, wherein the propagation module is 
further for 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

17. (Original) The image processing system of Claim 12, wherein the propagation module is 
further for propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel surrounded by a predefined size window in the continuous depth surface. 
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18. (Currently Amended) A computer readable medium including computer 
instructions for a 3-D image reconstruction computer system, the computer instructions 
comprising instructions for: 

detecting a plurality of pixels in a base image that represents a first view of a scene, 
wherein the first view is one of at least three relative views of a scene; 

determining 3-D depth of the plurality of pixels in the base image by contemporaneously 
matching image features between three or more relative images of a scene by matching 
correspondence to a plurality of pixels in three or more relative images representing the at least 
three relative views of the scene, wherein the three or more images are captured relative to each 
other and are of the same scene, and wherein each of the three or more images includes at least 
an overlapping portion of the scene relative to each other, wherein the at least three relative 
views of the scene are situated in a non-linear arrangement relative to each other and are further 
oriented in non-parallel planes relative to each other; and 

tracing pixels in a virtual piecewise continuous depth surface by spatial propagation 
starting from the detected pixels in the base image by using the matching and corresponding 
plurality of pixels in the three or more relative images to create the virtual piecewise continuous 
depth surface viewed from the base image, each successfully traced pixel being associated with a 
depth in the scene viewed from the base image. 
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19. (Previously Presented) The computer readable medium of claim 18, wherein the step of 
determining the 3-D depth comprises the steps of: 

detecting a plurality of image pixels in a first image corresponding to a first view of a 

scene; 

detecting a plurality of image pixels in at least a second image corresponding to a 
respective at least a second view of the scene, wherein the at least a second image deviates from 
the first image as a result of camera relative motion; 

determining a first two-view correspondence between the plurality of detected image 
pixels in the first image and a plurality of detected image pixels in one of the at least a second 
image resulting in a potential two-view match set of candidate image pixels between the first 
image and the one of the at least a second image; 

concatenating a plurality of two-view match sets to form a first multiple-view potential 
match set; and 

determining a second multiple-view potential match set that is a refinement of the first 
multiple-view potential match set. 

20. (Previously Presented) The computer readable medium of claim 19, wherein the second 
multiple-view potential match set is based at least in part on a least median of squares 
computation of the reprojection errors related to matched pixels in the first multiple-view 
potential match set. 

21. (Original) The computer readable medium of claim 18, wherein the tracing step comprises 
the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel starting from the detected pixels in the base image. 

22. (Original) The computer readable medium of Claim 21, wherein the tracing step comprises 
the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 
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23. (Original) The computer readable medium of Claim 21, wherein the tracing step comprises 
the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

24. (Original) The computer readable medium of Claim 18, wherein the tracing step comprises 
the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel surrounded by a predefined size window in the continuous depth surface. 

25. (Original) The computer readable medium of Claim 24, wherein the tracing step comprises 
the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 

26. (Original) The computer readable medium of Claim 24, wherein the tracing step comprises 
the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

27. (Original) The computer readable medium of Claim 26, wherein the matching cost is 
determined by computing a normalized cross-correlation between the windows associated with 
the two propagating fronts about the reached boundary. 

28. (Original) The computer readable medium of Claim 27, further comprising a step of: 

rectification of at least one pair of images corresponding to at least one pair of views of 
the scene. 
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29. (Currently Amended) A method of recovering depth information for pixels of a 
base image representing a view of a scene, the method comprising the steps of: 
capturing a base image representing a view of a scene; 

tracing at least one parameter surface associated with the base image, each of the at least 
one parameter surface traced starting from at least one predetermined seed pixel point associated 
with the base image; and 

calculating a derivative of function E(g) with respect to parameter g by using finite 
difference to minimize the following equation 

E(g) = f^{l-NCc[lMslv(g)),lSuj,v^)]}, 

where NCC[/; (m , v ), 1^ {uj , ) J is a normalized cross -correlation between /; (u, v ) and (Uj ,Vj); 
/; (u, V ) is a first window of size cox co centered at pixel (u,v) in 
Iq(Uj,Vj) is a second window of size coxco centered at pixel (uj,Vj) in Iq, 
li is a reference image, 
Iq is a base image. 

(w, V ) is a pixel point closest to Pi(Co+gL(uj, vj)); 

Co+gL(uj, Vj) is a 3-D point that projects to (uj, vj) in the base view, and to Pi(Co+gL(uj, vj)) in the 

reference view; 
Co is the base image camera's center of projection; 

L(uj,Vj) is the unit vector from Co to the point on the image plane that corresponds the pixel 

(uj,vj); 

^ is a depth parameter; and 

storing the calculated derivative of the function E(g) in a memory [[.]]iand 

generating a three-dimensional image of at least a portion of the scene using at least the 

calculated derivative of the function E(g) that has been stored in memory. 
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